Online Knowledge%2DBased Support Vector Machines.
Abstract. Prior knowledge, in the form of simple advice rules, can greatly speed up convergence in learning algorithms. Online learning methods predict the label of the current point and then receive the correct label (and learn from that information). The goal of this work is to update the hypothesis taking into account not just the label feedback, but also the prior knowledge, in the form of soft polyhedral advice, so as to make increasingly accurate predictions on subsequent examples. Advice helps speed up and bias learning so that generalization can be obtained with less data. Our passive-aggressive approach updates the hypothesis using a hybrid loss that takes into account the margins of both the hypothesis and the advice on the current point. Encouraging computational results and loss bounds are provided. 1
Transfer Learning via Advice Taking.
Abstract The goal of transfer learning is to speed up learning in a new task by transferring knowledge from one or more related source tasks. We describe a transfer method in which a reinforcement learner analyzes its experience in the source task and learns rules to use as advice in the target task. The rules, which are learned via inductive logic programming, describe the conditions under which an action is successful in the source task. The advice-taking algorithm used in the target task allows a reinforcement learner to benefit from rules even if they are imperfect. A human-provided mapping describes the alignment between the source and target tasks, and may also include advice about the differences between them. Using three tasks in the RoboCup simulated soccer domain, we demonstrate that this transfer method can speed up reinforcement learning substantially. 1

Rule Extraction for Transfer Learning.
Typically rule extraction is done for the purposes of human interpretation. However, there are other possible applications of rule extraction. One practical application is transfer learning, in which knowledge learned in one task is used to aid in learning a related task. The extracted rules, which explain the learned solution

Relational Macros for Transfer in Reinforcement Learning.
Abstract. We describe an application of inductive logic programming to transfer learning. Transfer learning is the use of knowledge learned in a source task to improve learning in a related target task. The tasks we work with are in reinforcement-learning domains. Our approach transfers relational macros, which are finite-state machines in which the transition conditions and the node actions are represented by first-order logical clauses. We use inductive logic programming to learn a macro that characterizes successful behavior in the source task, and then use the macro for decision-making in the early learning stages of the target task. Through experiments in the RoboCup simulated soccer domain, we show that Relational Macro Transfer via Demonstration (RMT-D) from a source task can provide a substantial head start in the target task. 1
Building Relational World Models for Reinforcement Learning.
Abstract. Many reinforcement learning domains are highly relational. While traditional temporal-difference methods can be applied to these domains, they are limited in their capacity to exploit the relational nature of the domain. Our algorithm, AMBIL, constructs relational world models in the form of relational Markov decision processes (MDPs). AMBIL works backwards from collections of high-reward states, utilizing inductive logic programming to learn their preimage, logical definitions of the region of state space that leads to the high-reward states via some action. These learned preimages are chained together to form an MDP that abstractly represents the domain. AMBIL estimates the reward and transition probabilities of this MDP from past experience. Since our MDPs are small, AMBIL uses valueiteration to quickly estimate the Q-values of each action in the induced states and determine a policy. AMBIL is able to employ complex background knowledge and supports relational representations. Empirical evaluation on both synthetic domains and a sub-task of the RoboCup soccer domain shows significant performance gains compared to standard Q-learning. 1
A Simple and Effective Method for Incorporating Advice into Kernel Methods.
We propose a simple mechanism for incorporating advice (prior knowledge), in the form of simple rules, into support-vector methods for both classification and regression. Our approach is based on introducing inequality constraints associated with datapoints that match the advice. These constrained datapoints can be standard examples in the training set, but can also be unlabeled data in a semi-supervised, advice-taking approach. Our new approach is simpler to implement and more efficiently solved than the knowledge-based support vector classification methods of Fung, Mangasarian and Shavlik (2002; 2003) and the knowledge-based support vector regression method of Mangasarian, Shavlik, and Wild (2004), while performing approximately as well as these more complex approaches. Experiments using our new approach on a synthetic task and a reinforcementlearning problem within the RoboCup soccer simulator show that our advice-taking method can significantly outperform a method without advice and perform similarly to prior advice-taking, support-vector machines.
An End%2Dto%2DEnd Supervised Target%2DWord Sense Disambiguation System.
We present an extensible supervised Target-Word Sense Disambiguation system that leverages upon GATE
Kernel Methods for Word Sense Disambiguation and Acronym Expansion.

Skill Acquisition Via Transfer Learning and Advice Taking.
Abstract. We describe a reinforcement learning system that transfers skills from a previously learned source task to a related target task. The system uses inductive logic programming to analyze experience in the source task, and transfers rules for when to take actions. The target task learner accepts these rules through an advice-taking algorithm, which allows learners to benefit from outside guidance that may be imperfect. Our system accepts a human-provided mapping, which specifies the similarities between the source and target tasks and may also include advice about the differences between them. Using three tasks in the RoboCup simulated soccer domain, we demonstrate that this system can speed up reinforcement learning substantially. 1
Giving Advice about Preferred Actions to Reinforcement Learners Via Knowledge%2DBased Kernel Regression.
We present a novel formulation for providing advice to a reinforcement learner that employs supportvector regression as its function approximator. Our new method extends a recent advice-giving technique, called Knowledge-Based Kernel Regression (KBKR), that accepts advice concerning a single action of a reinforcement learner. In KBKR, users can say that in some set of states, an action’s value should be greater than some linear expression of the current state. In our new technique, which we call Preference KBKR (Pref-KBKR), the user can provide advice in a more natural manner by recommending that some action is preferred over another in the specified set of states. Specifying preferences essentially means that users are giving advice about policies rather than Q values, which is a more natural way for humans to present advice. We present the motivation for preference advice and a proof of the correctness of our extension to KBKR. In addition, we show empirical results that our method can make effective use of advice on a novel reinforcement-learning task, based on the RoboCup simulator, which we call Breakaway. Our work demonstrates the significant potential of advice-giving techniques for addressing complex reinforcement learning problems, while further demonstrating the use of support-vector regression for reinforcement learning.
Using Advice to Transfer Knowledge Acquired in One Reinforcement Learning Task to Another.
Abstract. We present a method for transferring knowledge learned in one task to a related task. Our problem solvers employ reinforcement learning to acquire a model for one task. We then transform that learned model into advice for a new task. A human teacher provides a mapping from the old task to the new task to guide this knowledge transfer. Advice is incorporated into our problem solver using a knowledge-based support vector regression method that we previously developed. This advice-taking approach allows the problem solver to refine or even discard the transferred knowledge based on its subsequent experiences. We empirically demonstrate the effectiveness of our approach with two games from the RoboCup soccer simulator: KeepAway and BreakAway. Our results demonstrate that a problem solver learning to play BreakAway using advice extracted from KeepAway outperforms a problem solver learning without the benefit of such advice. 1
A Comparative Study of Support Vector Machines Applied to the Supervised Word Sense Disambiguation Problem in the Medical Domain.
Abstract. We have applied five supervised learning approaches to word sense disambiguation in the medical domain. Our objective is to evaluate Support Vector Machines (SVMs) in comparison with other well known supervised learning algorithms including the naïve Bayes classifier, C4.5 decision trees, decision lists and boosting approaches. Based on these results we introduce further refinements of these approaches. We have made use of unigram and bigram features selected using different frequency cut-off values and window sizes along with the statistical significance test of the log likelihood measure for bigrams. Our results show that overall, the best SVM model was most accurate in 27 of 60 cases, compared to 22, 14, 10 and 14 for the naïve Bayes, C4.5 decision trees, decision list and boosting methods respectively.
Knowledge transfer via advice taking.

Exploiting unlabeled data in ensemble methods.
An adaptive semi-supervised ensemble method, ASSEMBLE, is proposed that constructs classification ensembles based on both labeled and unlabeled data. ASSEMBLE alternates between assigning &#034;pseudo-classes&#034; to the unlabeled data using the existing ensemble and constructing the next base classifier using both the labeled and pseudolabeled data. Mathematically, this intuitive algorithm corresponds to maximizing the classification margin in hypothesis space as measured on both the labeled and unlabeled data. Unlike alternative approaches, ASSEMBLE does not require a semi-supervised learning method for the base classifier. ASSEMBLE can be used in conjunction with any cost-sensitive classification algorithm for both two-class and multi-class problems. ASSEMBLE using decision trees won the NIPS 2001 Unlabeled Data Competition. In addition, strong results on several benchmark datasets using both decision trees and neural networks support the proposed method.
Popular Ensemble Methods An Empirical Study.
An ensemble consists of a set of individually trained classifiers (such as neural networks or decision trees) whose predictions are combined when classifying novel instances. Previous research has shown that an ensemble is often more accurate than any of the single classifiers in the ensemble. Bagging (Breiman, 1996c) and Boosting (Freund &amp; Schapire, 1996; Schapire, 1990) are two relatively new but popular methods for producing ensembles. In this paper we evaluate these methods on 23 data sets using both neural networks and decision trees as our classification algorithm. Our results clearly indicate a number of conclusions. First, while Bagging is almost always more accurate than a single classifier, it is sometimes much less accurate than Boosting. On the other hand, Boosting can create ensembles that are less accurate than a single classifier – especially when using neural networks. Analysis indicates that the performance of the Boosting methods is dependent on the characteristics of the data set being examined. In fact, further results show that Boosting ensembles may overfit noisy data sets, thus decreasing its performance. Finally, consistent with previous studies, our work suggests that most of the gain in an ensemble’s performance comes in the first few classifiers combined; however, relatively large gains can be seen up to 25 classifiers when Boosting decision trees. 1.
Boosting Classifiers Regionally.
This paper presents a new algorithm for Boosting  the performance of an ensemble of classifiers. In  Boosting, a series of classifiers is used to predict  the class of data where later members of the series  concentrate on training data that is incorrectly  predicted by earlier members. To make a prediction  about a new pattern, each classifier predicts  the class of the pattern and these predictions are  then combined. In standard Boosting, the predictions  are combined by weighting the predictions  by a term related to the accuracy of the classifier  on the training data. This approach ignores the  fact that later classifiers focus on small subsets  of the patterns and thus may only be good at  classifying similar patterns. In RegionBoost, this  problem is addressed by weighting each classifier  &#039;s predictions by a factor measuring how well  that classifier performs on similar patterns. In  this paper we examine several methods for determining  how well a classifier performs on simil...
An Empirical Evaluation of Bagging and Boosting.
An ensemble consists of a set of independently trained classi ers (such as neural networks or decision trees) whose predictions are combined when classifying novel instances. Previous research has shown that an ensemble as a whole is often more accurate than any of the single classiers in the ensemble. Bagging (Breiman 1996a) and Boosting (Freund &amp; Schapire 1996) are two relatively new but popular methods for producing ensembles. In this paper we evaluate these methods using both neural networks and decision trees as our classi cation algorithms. Our results clearly showtwo important facts. The rst is that even though Bagging almost always produces a better classi er than any of its individual component classi ers and is relatively impervious to over tting, it does not generalize any better than a baseline neural-network ensemble method. The second is that Boosting is apowerful technique that can usually produce better ensembles than Bagging � however, it is more susceptible to noise and can quickly over t a data set.
Feature Engineering and Classifier Selection A Case Study in Venusian Volcano Detection.
As machine learning has graduated from  toy problems to &#034;real world&#034; applications,  users are finding that &#034;real world&#034; problems  require them to perform aspects of problem  solving that are not currently addressed  by much of the machine learning literature.  Specifically, users are finding that the tasks of  selecting a set of features to define a problem  and obtaining a set of examples of the problem  are often more important for a successful  machine learning application than the selection  or development of a specific classification  method. In this paper we present a case  study of machine learning applied to a difficult  &#034;real world&#034; problem: detecting volcanos  in SAR (synthetic aperture radar) images of  Venus from the Magellan dataset. Our work  demonstrates that the processes of feature  selection and sample collection are critical  to the production of a good classifier. We  further show that the use of domain dependent  knowledge can often serve to enhance  the resulting ...
Ensembles as a Sequence of Classifiers.
An ensemble is a classifier created by combining the predictions of multiple component classifiers. We present a new method for combining classifiers into an ensemble based on a simple estimation of each classifier&#039;s competence. The classifiers are grouped into an ordered list where each classifier has a corresponding threshold. To classify an example, the first classifier on the list is consulted and if that classifier&#039;s confidence for predicting the example is above the classifier&#039;s threshold, then that classifier&#039;s prediction is used. Otherwise, the next classifier and its threshold is consulted and so on. If none of the classifiers predicts the example above its confidence threshold then the class of the example is predicted by averaging all of the component classifier predictions. The key to this method is the selection of the confidence threshold for each classifier. We have implemented this method in a system called SEQUEL which has been applied to the task of recognizing volcanos in SAR images of Venus. In this domain, SEQUEL outperforms each individual classifier as well as the simple approach of using an ensemble constructed from the average prediction of all the classifiers. 1
Combining the Predictions of Multiple Classifiers Using Competitive Learning to Initialize Neural Networks.
The primary goal of inductive learning is to generalize well -- that is, induce a function that accurately produces the correct output for future inputs. Hansen and Salamon showed that, under certain assumptions, combining the predictions of several separately trained neural networks will improve generalization. One of their key assumptions is that the individual networks should be independent in the errors they produce. In the standard way of performing backpropagation this assumption may be violated, because the standard procedure is to initialize network weights in the region of weight space near the origin. This means that backpropagation&#039;s gradient-descent search may only reach a small subset of the possible local minima. In this paper we present an approach to initializing neural networks that uses competitive learning to intelligently create networks that are originally located far from the origin of weight space, thereby potentially increasing the set of reachable local minima....
Creating Advice%2DTaking Reinforcement Learners.
. Learning from reinforcements is a promising approach for creating intelligent agents. However, reinforcement learning usually requires a large number of training episodes. We present and evaluate a design that addresses this shortcoming by allowing a connectionist Q-learner to accept advice given, at any time and in a natural manner, by an external observer. In our approach, the advice-giver watches the learner and occasionally makes suggestions, expressed as instructions in a simple imperative programming language. Based on techniques from knowledge-based neural networks, we insert these programs directly into the agent&#039;s utility function. Subsequent reinforcement learning further integrates and refines the advice. We present empirical evidence that investigates several aspects of our approach and show that, given good advice, a learner can achieve statistically significant gains in expected reward. A second experiment shows that advice improves the expected reward regardless of the...
Combining the Predictions of Multiple Classifiers Using Competitive Learning to Initialize Neural Networks.
The primary goal of inductive learning is to generalize well -- that is, induce a function that accurately produces the correct output for future inputs. Hansen and Salamon showed that, under certain assumptions, combining the predictions of several separately trained neural networks will improve generalization. One of their key assumptions is that the individual networks should be independent in the errors they produce. In the standard way of performing backpropagation this assumption may be violated, because the standard procedure is to initialize network weights in the region of weight space near the origin. This means that backpropagation&#039;s gradient-descent search may only reach a small subset of the possible local minima. In this paper we present an approach to initializing neural networks that uses competitive learning to intelligently create networks that are originally located far from the origin of weight space, thereby potentially increasing the set of reachable local minima....
Incorporating Advice into Agents that Learn from Reinforcements.
Learning from reinforcements is a promising approach for creating intelligent agents. However, reinforcement learning usually requires a large number of training episodes. We present an approach that addresses this shortcoming by allowing a connectionist Q-learner to accept advice given, at any time and in a natural manner, by an external observer. In our approach, the advice-giver watches the learner and occasionally makes suggestions, expressed as instructions in a simple programming language. Based on techniques from knowledge-based neural networks, these programs are inserted directly into the agent&#039;s utility function. Subsequent reinforcement learning further integrates and refines the advice. We present empirical evidence that shows our approach leads to statistically-significant gains in expected reward. Importantly, the advice improves the expected reward regardless of the stage of training at which it is given.  Introduction  A successful and increasingly popular method for cr...
Using Knowledge%2DBased Neural Networks to Improve Algorithms Refining the Chou%2DFasman Algorithm for Protein Folding.
We describe a method for using machine learning to refine algorithms represented as generalized finite-state automata. The knowledge in an automaton is translated into an artificial neural network, and then refined with backpropagation on a set of examples. Our technique for translating an automaton into a network extends  kbann, a system that translates a set of propositional rules into a corresponding neural network. The extended system, FSkbann, allows one to refine the large class of algorithms that can be represented as state-based processes. As a test, we use FSkbann  to refine the Chou-Fasman algorithm, a method for predicting how globular proteins fold. Empirical evidence shows the refined algorithm FSkbann produces is statistically significantly more accurate than both the original Chou-Fasman algorithm and a neural network trained using the standard approach.  Introduction  As machine learning has been increasingly applied to complex real-world problems, many researchers have...
Using Knowledge%2DBased Neural Networks to Improve Algorithms Refining the Chou%2DFasman Algorithm for Protein Folding.
We describe a method for using machine learning to refine algorithms represented as generalized finite-state automata. The knowledge in an automaton is translated into an artificial neural network, and then refined with backpropagation on a set of examples. Our technique for translating an automaton into a network extends  kbann, a system that translates a set of propositional rules into a corresponding neural network. The extended system, FSkbann, allows one to refine the large class of algorithms that can be represented as state-based processes. As a test, we use FSkbann  to refine the Chou-Fasman algorithm, a method for predicting how globular proteins fold. Empirical evidence shows the refined algorithm FSkbann produces is statistically significantly more accurate than both the original Chou-Fasman algorithm and a neural network trained using the standard approach.  Introduction  As machine learning has been increasingly applied to complex real-world problems, many researchers have...
Refining Domain Theories Expressed as Finite%2DState Automata.
Abstract The KBANN system uses neural networks torefine domain theories. Currently, domain knowledge in KBANN is expressed as non-recursive, propositional rules. We extend KBANN to domain theories expressed asfinite-state automata. We apply finite-state KBANN to the task of predicting how proteinsfold, producing a small but statistically significant gain in accuracy over both astandard neural network approach and a non-learning algorithm from the biologicalliterature. Our method shows promise at solving this and other real-world problemsthat can be described in terms of statedependent decisions. 1 INTRODUCTION Research in artificial neural networks (ANNs) has untilrecently largely ignored preexisting knowledge about the
Enriching Vocabularies by Generalizing Explanation Structures.

